Motion and context sharing for pen-based computing inputs

ABSTRACT

A “Motion and Context Sharing Technique” uses a pen or stylus enhanced to incorporate multiple sensors, i.e., a “sensor pen,” and a power supply to enable various input techniques and gestures. Various combinations of pen stroke, pressure, motion, and other sensor pen inputs are used to enable various hybrid input techniques that incorporate simultaneous, concurrent, sequential, and/or interleaved, sensor pen inputs and touch inputs (i.e., finger, palm, hand, etc.) on displays or other touch sensitive surfaces. This enables a variety of motion-gesture inputs relating to the context of how the sensor pen is used or held, even when the pen is not in contact or within sensing range of the computing device digitizer. In other words, any particular touch inputs or combinations of touch inputs are correlated with any desired sensor pen inputs, with those correlated inputs then being used to initiate any desired action by the computing device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-in-Part of a prior applicationentitled “MULTI-TOUCH INPUT DEVICE WITH ORIENTATION” by Xiang Cao, etal., which was filed with the USPTO on Feb. 11, 2011 and assigned Ser.No. 13/026,058, the subject matter of which is incorporated herein bythis reference.

BACKGROUND

Many mobile computing devices (e.g., tablets, phones, etc.) use a pen,pointer, or stylus type input device (collectively referred to herein asa “pen type input device” or “pen”) in combination with a digitizercomponent of the computing device for input purposes. Typically, pentype input devices enable a variety of multi-modal pen, touch, andmotion based input techniques.

Various conventional input techniques have adapted pen type devices toprovide auxiliary input channels including various combinations oftilting, rolling, and pressure sensing. However, one of the limitationsof many of these techniques is that they operate using sensors coupledto the computing device to sense and consider pen movements or hoverconditions that are required to be in close proximity to the digitizerso that the pen can be sensed by the digitizer. Many such techniquesoperate in a context where the pen is used to perform various inputactions that are then sensed and interpreted by the computing device.

For example, one conventional technique considers pen rolling duringhandwriting and sketching tasks, as well as various intentional penrolling gestures. However, these pen rolling techniques operate in closeproximity to the computing device based on sensors associated with thecomputing device. Related techniques that require the pen type inputdevice to maintain contact (or extreme proximity) with the digitizerinclude various tilt and pressure based pen inputs. Various examples ofsuch techniques consider separate or combined tilt and pressure inputsin various tablet-based settings for interacting with context menus,providing multi-parameter selection, object or menu manipulation, widgetcontrol, etc.

In contrast, various conventional techniques use anaccelerometer-enhanced pen to sense movements when the pen or stylus isnot touching the display. The sensed movements are then provided to thecomputing device for input purposes such as shaking the stylus to cyclethrough color palettes, and rolling the stylus to pick colors or scrollweb pages. A somewhat related technique provides a pointing devicehaving multiple inertial sensors to enable three-dimensional pointing ina “smart room” environment. This technique enables a user to gesture toobjects in the room and speak voice commands. Other techniques use 3Dspatial input to employ stylus-like devices in free space, but requireabsolute tracking technologies that are generally impractical for mobilepen-and-tablet type interactions.

Recently various techniques involving the use of contact (touch) sensorsor multi-contact pressure (non-zero force) sensors on a pen surface havebeen used to enable various grip-sensing input scenarios. For example,stylus barrels have been developed to provide multi-touch capabilitiesfor sensing finger gestures. Specific grips can also be associated withparticular pens or brushes. Several conventional systems employ inertialsensors in tandem with grip sensing to boost grip pattern recognition.

Further, various conventional systems combine pen tilt with direct-touchinput. One such system uses a stylus that senses which corners, edges,or sides of the stylus come into contact with a tabletop display. Thus,by tilting or rolling the stylus while it remains in contact with thedisplay, the user can fluidly switch between a number of tools, modes,and other input controls. This system also combines direct multi-touchinput with stylus orientation, allowing users to tap a finger on acontrol while holding or “tucking” the stylus in the palm. However, thissystem requires contact with the display in order to sense tilt or othermotions. Related techniques combine both touch and motion for mobiledevices by using direct touch to cue the system to recognize shaking andother motions of pen type input devices.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. Further, while certain disadvantages of prior technologies maybe noted or discussed herein, the claimed subject matter is not intendedto be limited to implementations that may solve or address any or all ofthe disadvantages of those prior technologies.

In general, a “Motion and Context Sharing Technique,” as describedherein, provides a variety of input techniques based on variouscombinations of pen input, direct-touch input, and motion-sensinginputs. In contrast to existing pen based input techniques, the Motionand Context Sharing Techniques described herein leverage inputs fromsome or all of the sensors of a “sensor pen” in combination withdisplays or other surfaces that support both pen or stylus inputs anddirect multi-touch input. In other words, various embodiments of theMotion and Context Sharing Technique consider various combinations ofpen stroke, pressure, motion, and other inputs in the context oftouch-sensitive displays, in combination with various hybrid techniquesthat incorporate simultaneous, concurrent, sequential, and/orinterleaved, sensor pen inputs and touch inputs (i.e., finger, palm,hand, etc.) on the display or other touch sensitive surface of thecomputing device.

Note that the term pressure, as relating to pressure sensors and thelike may refer to various sensor types and configurations. For example,in various cases and embodiments, pressure may refer to pen tip pressureexerted on a display. In general, pen tip pressure is typically sensedby some type of pressure transducer inside the pen, but it is alsopossible to have the pen tip pressure sensing done by thedisplay/digitizer itself in some devices. In addition, the term pressureor pressure sensing or the like may also refer to a separate channel ofsensing the grip pressure of the hand (or fingers) contacting anexterior casing or surface of the pen. Various sensing modalitiesemployed by the Motion and Context Sharing Technique may separately orconcurrently consider or employ both types of pressure sensing (i.e.,pen tip pressure and pen grip pressure) for initiating various motiongestures.

Note that various devices used to enable some of the many embodiments ofa “Motion and Context Sharing Technique,” as described herein, includepens, pointers, stylus type input devices, etc., that are collectivelyreferred to herein as a “sensor pen” for purposes of discussion. Notealso that the functionality described herein may be implemented in anydesired form factor, e.g., wand, staff, ball racquet, toy sword, etc.,for use with various gaming devices, gaming consoles, or other computingdevices. Further, the sensor pens described herein are adapted toincorporate various combinations of a power supply and multiple sensorsincluding, but not limited to inertial sensors, accelerometers, pressuresensors, grip sensors, near-field communication sensors, RFID tagsand/or sensors, temperature sensors, microphones, magnetometers,capacitive sensors, gyroscopes, etc., in combination with variouswireless communications capabilities for interfacing with variouscomputing devices. In addition, in various embodiments, the sensor pensdescribed herein have been further adapted to incorporate digital memoryand/or computing capabilities that allow the sensor pens to act incombination or cooperation with other computing devices, other sensorpens, or even as a standalone computing device.

Advantageously, the various embodiments of the Motion and ContextSharing Technique described herein use various wired and/or wirelesscommunication techniques integrated into the sensor pen to enable inputsand gestures that are not restricted to a near-proximity sensing rangeof the digitizer of the computing device. In addition, another advantageof the Motion and Context Sharing Technique described herein is that theuse of a wide range of sensors and a communication interface in thesensor pen enables a wide array of sensing dimensions that provide newinput scenarios and gestures for computing devices. Examples inputscenarios include, but are not limited to, using electronically activesensor pens for tablets, electronic whiteboards, or other direct-inputdevices since the sensor pen itself integrates motion and/orgrip-sensing and other sensor-based capabilities that allow richerin-air pen-based gestures at arbitrary distances from the computingdevice, as well as richer sensing of user context information.

Further, these capabilities enable realization of a wide range of newpen-based gestures and input scenarios, many of which are simply notsupported by conventional tablet-digitizer technologies due to thenecessity of conventional pens to be extremely close to the display forthe digitizer to sense the pens presence. In addition, given the sensorsand communications capabilities of the sensor pen, the Motion andContext Sharing Technique provides various mechanisms that can be usedto optimize the behavior of computing devices and user experience basedon concurrently sensing input states of the sensor pen and the computingdevice. Simple examples of this concept include, but are not limited to,alerting the user to a forgotten sensor pen, for example, as well assensing whether the user is touching the display with a hand that isalso grasping a sensor pen. This can be used, for example, to makefine-grained distinction among touch gestures as well as to support avariety of “palm rejection” techniques for eliminating or avoidingunintentional touch inputs.

In view of the above summary, it is clear that the Motion and ContextSharing Technique described herein uses a sensor pen to enable a varietyof input techniques and gestures based on various combinations ofdirect-touch inputs and sensor pen inputs that are not restricted to anear-proximity sensing range of the digitizer of a computing device. Inaddition to the just described benefits, other advantages of the Motionand Context Sharing Technique will become apparent from the detaileddescription that follows hereinafter when taken in conjunction with theaccompanying drawing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the claimed subjectmatter will become better understood with regard to the followingdescription, appended claims, and accompanying drawings where:

FIG. 1 shows a general operational overview of a “Motion and ContextSharing Technique” that illustrates interoperation between a sensor penand a touch-sensitive computing for triggering one or more motiongestures or other actions, as described herein.

FIG. 2 provides an exemplary architectural flow diagram that illustratesprogram modules for implementing various embodiments of the Motion andContext Sharing Technique, as described herein.

FIG. 3 provides an illustration of using the Motion and Context SharingTechnique to provide a correlated touch and sensor pen input mechanism,as described herein.

FIG. 4 provides an illustration of using the Motion and Context SharingTechnique to provide a roll to undo input mechanism, as describedherein.

FIG. 5 provides an illustration of using the Motion and Context SharingTechnique to provide a finger tap input mechanism, as described herein.

FIG. 6 provides an illustration of using the Motion and Context SharingTechnique to provide a touch and spatter input mechanism for painting,drawing, or sketching type applications, as described herein.

FIG. 7 provides an illustration of using the Motion and Context SharingTechnique to provide a touch and tilt for layers input mechanism, asdescribed herein.

FIG. 8 provides an illustration of using the Motion and Context SharingTechnique to provide a touch and roll to rotate input mechanism, asdescribed herein.

FIG. 9 provides an illustration of using the Motion and Context SharingTechnique to provide a vertical menu input mechanism, as describedherein.

FIG. 10 provides an illustration of using the Motion and Context SharingTechnique to provide a hard tap input mechanism, as described herein.

FIG. 11 illustrates a general system flow diagram that illustratesexemplary methods for implementing various embodiments of the Motion andContext Sharing Technique, as described herein.

FIG. 12 is a general system diagram depicting a simplifiedgeneral-purpose computing device having simplified computing and I/Ocapabilities, in combination with a sensor pen having various sensors,power and communications capabilities, for use in implementing variousembodiments of the Motion and Context Sharing Technique, as describedherein.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In the following description of the embodiments of the claimed subjectmatter, reference is made to the accompanying drawings, which form apart hereof, and in which is shown by way of illustration specificembodiments in which the claimed subject matter may be practiced. Itshould be understood that other embodiments may be utilized andstructural changes may be made without departing from the scope of thepresently claimed subject matter.

1.0 Introduction:

In general, a “Motion and Context Sharing Technique,” as describedherein, provides various techniques for using a pen or stylus enhancedwith a power supply and multiple sensors, i.e., a “sensor pen,” toenable a variety of input techniques and gestures. These techniquesconsider various combinations of pen stroke, pressure, motion, and otherinputs in the context of touch-sensitive displays, in combination withvarious hybrid techniques that incorporate simultaneous, concurrent,sequential, and/or interleaved, sensor pen inputs and touch inputs(i.e., finger, palm, hand, etc.) on displays or other touch sensitivesurfaces. These techniques enable a wide variety of motion-gestureinputs, as well sensing the context of how a user is holding or usingthe sensor pen, even when the pen is not in contact, or even withinsensing range, of the digitizer of a touch sensitive computing device.

Note that the term pressure, as relating to pressure sensors and thelike may refer to various sensor types and configurations. For example,in various cases and embodiments, pressure may refer to pen tip pressureexerted on a display. In general, pen tip pressure is typically sensedby some type of pressure transducer inside the pen, but it is alsopossible to have the pen tip pressure sensing done by thedisplay/digitizer itself in some devices. In addition, the term pressureor pressure sensing or the like may also refer to a separate channel ofsensing the grip pressure of the hand (or fingers) contacting anexterior casing or surface of the pen. Various sensing modalitiesemployed by the Motion and Context Sharing Technique may separately orconcurrently consider or employ both types of pressure sensing (i.e.,pen tip pressure and pen grip pressure) for initiating various motiongestures.

Note that various devices used to enable some of the many embodiments ofthe “Motion and Context Sharing Technique” include pens, pointers,stylus type input devices, etc., that are collectively referred toherein as a “sensor pen” for purposes of discussion. Note also that thefunctionality described herein may be implemented in any desired formfactor, e.g., wand, staff, ball racquet, toy sword, etc., for use withvarious gaming devices, gaming consoles, or other computing devices.Further, the sensor pens described herein are adapted to incorporate apower supply and various combinations of sensors including, but notlimited to inertial sensors, accelerometers, pressure sensors, gripsensors, near-field communication sensors, RFID tags and/or sensors,temperature sensors, microphones, magnetometers, capacitive sensors,gyroscopes, etc., in combination with various wireless communicationscapabilities for interfacing with various computing devices. Note thatany or all of these sensors may be multi-axis or multi-position sensors(e.g., 3-axis accelerometers, gyroscopes, and magnetometers). Inaddition, in various embodiments, the sensor pens described herein havebeen further adapted to incorporate memory and/or computing capabilitiesthat allow the sensor pens to act in combination or cooperation withother computing devices, other sensor pens, or even as a standalonecomputing device.

The Motion and Context Sharing Technique is adaptable for use with anytouch-sensitive computing device having one or more touch-sensitivesurfaces or regions (e.g., touch screen, touch sensitive bezel or case,sensors for detection of hover-type inputs, optical touch sensors,etc.). Note that touch-sensitive computing devices include both single-and multi-touch devices. Examples of touch-sensitive computing devicesinclude, but are not limited to, touch-sensitive display devicesconnected to a computing device, touch-sensitive phone devices,touch-sensitive media players, touch-sensitive e-readers, notebooks,netbooks, booklets (dual-screen), tablet type computers, or any otherdevice having one or more touch-sensitive surfaces or input modalities.Note also that the touch-sensitive region of such computing devices neednot be associated with a display, and furthermore that the location ortype of contact-sensitive region (e.g. front of a device on the display,vs. back of device without any associated display) may be considered asan input parameter for initiating one or more motion gestures (i.e.,user interface actions corresponding to the motion gesture).

Note also that the term “touch,” as used throughout this document willgenerally refer to physical user contact (e.g., finger, palm, hand,etc.) on touch sensitive displays or other touch sensitive surfaces of acomputing device using capacitive sensors or the like. Note also thatvirtual touch inputs relative to projected displays, electronicwhiteboards, or other surfaces or objects are treated by the Motion andContext Sharing Technique in the same manner as actual touch inputs on atouch-sensitive surface. Such virtual touch inputs are detected usingconventional techniques such as, for example, using cameras or otherimaging technologies to track user finger movement relative to aprojected image, relative to text on an electronic whiteboard, relativeto physical objects, etc.

In addition, it should be understood that the Motion and Context SharingTechnique is operable with a wide variety of touch and flex-sensitivematerials for determining or sensing touch or pressure. For example, onetouch-sensing technology adapted for use by the Motion and ContextSharing Technique determines touch or pressure by evaluating a lightsource relative to some definite deformation of a touched surface tosense contact. Also, note that sensor pens, as discussed herein mayinclude multiple types of touch and/or pressure sensing substrates. Forexample, sensor pens may be both touch-sensitive and/or pressuresensitive using any combination of sensors, such as, for example,capacitive sensors, pressure sensors, flex- or deformation-basedsensors, etc.

In addition, the Motion and Context Sharing Technique uses a variety ofknown techniques to for differentiating between valid and invalidtouches received by one or more touch-sensitive surfaces of thetouch-sensitive computing device. Examples of valid touches and contactsinclude single, simultaneous, concurrent, sequential, and/or interleaveduser finger touches (including gesture type touches), pen or stylustouches or inputs, hover-type inputs, or any combination thereof. Withrespect to invalid or unintended touches, the Motion and Context SharingTechnique disables or ignores one or more regions or sub-regions oftouch-sensitive input surfaces that are expected to receiveunintentional contacts, or intentional contacts not intended as inputs,for device or application control purposes. Examples of contacts thatmay not intended as inputs include, but are not limited to, a user'spalm resting on a touch screen while the user writes on that screen witha stylus or pen, holding the computing device by gripping a touchsensitive bezel, etc.

Further, the terms “contact” or “pen input” as used herein generallyrefer to interaction involving physical contact (or hover) of the sensorpen with a touch sensitive surface or digitizer component of thecomputing device. Note also that as discussed herein, inputs provided byone or more sensors of the sensor pen will generally be referred toherein as a “sensor input,” regardless of whether or not the sensor penis within a digitizer range or even in contact with a touch sensitivesurface or digitizer component of the computing device.

Consequently, it should be understood that any particular motiongestures or inputs described herein are derived from variouscombinations of simultaneous, concurrent, sequential, and/or interleavedpen inputs, user touches, and sensor inputs. It should be furtherunderstood that the current context or state of either or both thesensor pen and computing device is also considered when determiningwhich motion gestures or inputs to activate or initialize.

The Motion and Context Sharing Technique described herein provides anumber of advantages relating to pen or stylus based user interactionwith touch-sensitive computing devices, including, but not limited to:

-   -   Adopting the perspective of context-sensing from mobile        computing and applying it to motions of the stylus or sensor pen        itself;    -   Leveraging the ability to sense tilting or other explicit motion        gestures that occur close to the computing device as well as        beyond the hover-sensing range of the digitizer;    -   Integrating both pen motions and computing device motions for        initiating various actions relative to the context between the        sensor pen and computing device. Simple examples of the use of        such contextual information include the pen loss prevention        techniques described herein;    -   Providing expressive application-specific gestures such as a        “touch and spatter” technique for interacting with painting-type        applications, as well as more generic interactions such as        having the computing device respond to picking up and putting        down the pen (e.g., exit or enter sleep mode, open context        sensitive menu for a currently active object, etc.), or        cross-application gestures such as a “vertical menu” input        mechanism and support for application undo and redo sequences;        and    -   Combining pen stroke, finger touch, and motion-sensing together        into hybrid, multi-modal techniques that enable new types of        gestures for pen-operated tablet computers and other        touch-sensitive computing devices.

1.1 System Overview:

The Motion and Context Sharing Technique operates, in part, byconsidering motion-based inputs from the sensor pen to trigger variousactions with respect to the computing device. FIG. 1, provides a generaloperational overview of the Motion and Context Sharing Technique,illustrating interoperation between the sensor pen and the computingdevice to trigger one or more motion gestures or other actions. Morespecifically, FIG. 1 shows sensor pen 100 in communication with touchsensitive computing device 105 via communications link 110. As discussedin further detail herein, the sensor pen 100 includes a variety ofsensors. A sensor module 115 in the sensor pen 100 monitors readings ofone or more of those sensors, and provides them to a communicationsmodule 120 to be sent to the computing device 105.

The touch sensitive computing device 105 includes a sensor input module125 that receives input from one or more sensors of sensor pen (e.g.,inertial, accelerometers, pressure, grip, near-field communication,RFID, temperature, microphones, magnetometers, capacitive sensors,gyroscopes, etc.) and provides that input to a gesture activation module135. In addition, the gesture activation module 135 also receives inputfrom a touch input module 130 that receives input from user touch of oneor more touch sensitive surfaces of the computing device 105. Given thesensor inputs and the touch inputs, if any, the gesture activationmodule 135 then evaluate simultaneous, concurrent, sequential, and/orinterleaved sensor pen 100 inputs and touch inputs (i.e., finger, palm,hand, etc.) on displays or other touch sensitive surfaces of thecomputing device 105 relative to contexts of sensor pen and computingdevice to trigger or activate one or more motion gestures (e.g., motiongestures 140 through 170, discussed in further detail herein).

More specifically, the Motion and Context Sharing Technique sensesvarious properties of the sensor pen relative to various distancesbetween the sensor pen and the computing device (i.e., contact, hoverrange, and beyond hover range), and whether the motions of the sensorpen are correlated with a concurrent user touch of a display or someother touch-sensitive surface of the computing device or with somemotion of the computing device. These sensed properties of the sensorpen are then correlated with various touches or motions of the computingdevice, and may also be considered in view of the current contexts ofeither or both the sensor pen and computing device (e.g., whether theyare being held, moving, power state, application status, etc.), and usedto trigger a variety of “motion gestures” or other actions.

With respect to hover range, in various embodiments, the Motion andContext Sharing Technique considers distance of the sensor pen above thedigitizer of the computing device. While a variety of ranges can beconsidered, in various tested embodiments, three range categories wereconsidered, including: physical contact, within hover range of thedigitizer, or beyond range of the digitizer. The activation mechanismfor any particular motion gestures may consider these different rangesof the sensor pen, in combination with any other correlated inputs,touches, and/or motions of the computing device.

In general, motion gestures can be grouped into various categories.Examples include motions that employ device orientation (of either orboth the sensor pen and the computing device), whether absolute orrelative, versus other motion types, which can be used to groupdifferent styles of motion input including hard contact forces, sensingparticular patterns or gestures of movement, as well as techniques thatuse stability (e.g., the absence of motion or other particular sensorinputs) to trigger actions or specific contexts.

Many different motion gestures and input scenarios are enabled by theMotion and Context Sharing Technique include. Further, a wide variety ofuser-definable motion gestures and input scenarios are enabled byallowing the user to associate any desired combination of touches,contacts, and/or sensor pen motions with any desired action by thecomputing device. In other words, any particular touch inputs orcombinations of touch inputs are correlated with any desired sensorinputs and/or contacts, with those correlated inputs then being used toinitiate any desired action by the computing device.

Note that raw sensor readings can be reported or transmitted from thesensor pen to the computing device for evaluation and characterizationby the computing device. For example, raw sensor data from inertialsensors within the sensor pen can be reported by the sensor pen to thecomputing device, with the computing device then determining penorientation as a function of the data from the inertial sensors.Alternately, in various embodiments, the sensor pen uses onboardcomputational capability to evaluate the input from various sensors. Forexample, sensor data derived from inertial sensors within the sensor pencan be processed by a computational component of the sensor pen todetermine pen orientation, with the orientation of tilt then beingreported by the sensor pen to the computing device.

Clearly, any desired combination of reporting of raw sensor data andreporting of processed sensor data to the computing device by the sensorpen can be performed depending upon the computational capabilities ofthe sensor pen. However, for purposes of explanation, the followingdiscussion will generally refer to reporting of sensor data to thecomputing device by the sensor pen for further processing by thecomputing device to determine various motion gestures or other inputscenarios. A few examples of various motion gestures and input scenariosenabled by the Motion and Context Sharing Technique are brieflyintroduced below.

For example, one such input technique, referred to as a “touch and tiltfor layers” gesture, uses a concurrent user touch and sensor pen tilt toactivate or interact with different layers displayed on a screen. Notethat the touch and tilt for layers gesture is initiated with the sensorpen at any desired distance from the display. Sensor pen tilt isdetermined by one or more of the pen sensors and reported to thecomputing device via the communications capabilities of the sensor pen.The touch and tilt for layers gesture is discussed in further detailherein.

A related gesture referred to as a “touch and roll to rotate” gesture,uses a concurrent user touch on a displayed object (e.g., text, shape,image, etc.) and a user initiated sensor pen rolling motion to rotatethe touched object. Note that the touch and roll to rotate gesture isinitiated with the sensor pen at any desired distance from the display,with sensor pen tilt being determined by one or more of the pen sensorsand reported via the communications capabilities of the sensor pen. Thetouch and roll to rotate gesture is discussed in further detail herein.

Another gesture, referred to herein as a “roll to undo” gesture, usessensors of the pen to detect a user initiated rolling motion of thesensor pen, with the result being to undo previous operations,regardless of whether those actions were user initiated or automaticallyinitiated by the computing device or system. As with the touch and tiltfor layers gesture, the roll to undo gesture is initiated with thesensor pen at any desired distance from the display, with sensor penrolling motions being determined by one or more of the pen sensors andreported via the communications capabilities of the sensor pen. The rollto undo gesture is discussed in further detail herein.

Another gesture, referred to herein as a “vertical menu” gesture, usessensors of the pen to detect a user initiated motion of the sensor penthat occurs with the pen coming into proximity range (or in contact withthe screen) in an orientation approximately perpendicular relative tothe computing device. For example, in one embodiment, bringing the penclose to the screen with the pen in this perpendicular pose willinitiate opening or expansion of vertical software menu or the like,while motions moving away from the computing device or display (i.e.,motion beyond the proximity sensing range) may initiate closing orcontraction of the vertical software menu or the like. Note that thesemotions may act in concert with a cursor location on or near the menulocation at the time that the motion of the sensor pen is detected, orwith any other locus of interaction with the computing device. As withthe previously noted gestures, the vertical menu gesture is initiatedwith the sensor pen at any desired distance from the display, withsensor pen orientation and distance being determined by one or more ofthe pen sensors and reported via the communications capabilities of thesensor pen. Note that various additional embodiments and considerationswith respect to the vertical menu concept are discussed in furtherdetail in Section 2.8.1

Another gesture, referred to herein as a “touch and spatter” gesture,uses sensors of the pen to detect a user initiated rapping of the sensorpen motion while the user is touching the display surface of thecomputing device. In general, the touch and spatter gesture operates ina drawing or painting type application to initiate an action that mimicsthe effect of an artist rapping a loaded paint brush on her finger toproduce spatters of paint on the paper. In this case, the user touchesthe screen with a finger and then strikes the sensor pen against thatfinger (or any other finger, object, or surface). Note that, given thelimited hover-sensing range of typical tablets, the tablet typicallywill not know the actual (x, y) location of the pen tip. Consequently,the touch and spatter gesture initiates an action that produces spatters(in a currently selected pen color) centered on the finger contactpoint. As with the previously noted gestures, the touch and spattergesture is initiated with the sensor pen at any desired distance fromthe display, with sensor pen rapping motions being determined by one ormore of the pen sensors and reported via the communications capabilitiesof the sensor pen. The touch and spatter gesture is discussed in furtherdetail herein.

A similar gesture, referred to herein as a “barrel tap gesture” usessensors of the pen to detect a user initiated finger tap on the sensorpen while the user is holding that pen. In general, the barrel tapgesture operates at any desired distance from the computing device ordigitizer. More specifically, hard-contact finger taps on the barrel ofthe pen are used as a way to “replace” mechanical button input such as apen barrel button, mouse button, enter key (or other keyboard press), orother button associated with the locus of interaction between the userand the computing device. The sensor pen tap gestures are generallyidentified as acceleration spikes from accelerometers in or on thesensor pen, consistent with a finger strike on the pen barrel. Thebarrel tap gesture is discussed in further detail herein.

Another similar gesture, referred to herein as a “hard stroke gesture”uses sensors of the pen to detect a fast movement (i.e., accelerationbeyond a predetermined threshold) of the sensor pen when not contactingthe computing device. In general, the hard stroke gesture operates atany desired distance from the computing device or digitizer, but in someembodiments the gesture is accepted at the moment the pen physicallystrikes the digitizer screen, and may furthermore be registered when thepen strikes the screen within an acceptable range of relativeorientation to the tablet display. Such hard contact gestures with thescreen can be difficult to sense with traditional pressure sensors dueto sampling rate limitations, and because the relative orientation ofthe pen may not be known on traditional digitizers. More specifically,this particular sensor pen motion can be associated with any desiredaction of the computing device. In a tested embodiment, it was used toinitiate a lasso type operation in a drawing program. The hard strokegesture is discussed in further detail herein.

Other examples of correlated sensor pen motions relative to thecomputing device include using pen sensors (e.g., accelerometers,pressure sensors, inertial sensors, grip sensors, etc.) to determinewhen the sensor pen is picked up or put down by the user. By consideringthe current sensor pen context or state (i.e., picked up or put down)relative to a current context or state of the computing device (e.g.,held by the user, power off, etc.), any desired action can be initiated(e.g., exit sleep mode in computing device when pen picked up, or entersleep mode if pen set down).

Conversely, a similar technique considers motion of the computing devicerelative to the sensor pen. For example, if sensors in the computingdevice (e.g., accelerometers or other motion or positional sensors)indicate that the computing device is being held by a user that iswalking or moving when sensors in the pen indicate that the sensor penis stationary, an automated alert (e.g., visible, audible, tactile,etc.) is initiated by the computing device. Similarly, speakers orlights coupled to the sensor pen can provide any combination of visibleand audible alerts to alert the user that the computing device is movingwhile the sensor pen is stationary. These types of “pen loss prevention”techniques provide additional examples of using correlated motions (orother inputs) of the sensor pen relative to the computing device toinitiate various actions.

1.2 Configuration Overview:

As noted above, the “Motion and Context Sharing Technique,” providesvarious techniques for using a “sensor pen” to enable a variety of inputtechniques based on various combinations of direct-touch inputs andsensor pen inputs that are not restricted to a near-proximity sensingrange of the digitizer of a computing device. The processes summarizedabove are illustrated by the general system diagram of FIG. 2. Inparticular, the system diagram of FIG. 2 illustrates theinterrelationships between program modules for implementing variousembodiments of the Motion and Context Sharing Technique, as describedherein. Furthermore, while the system diagram of FIG. 2 illustrates ahigh-level view of various embodiments of the Motion and Context SharingTechnique, FIG. 2 is not intended to provide an exhaustive or completeillustration of every possible embodiment of the Motion and ContextSharing Technique as described throughout this document.

In addition, it should be noted that any boxes and interconnectionsbetween boxes that may be represented by broken or dashed lines in FIG.2 represent alternate embodiments of the Motion and Context SharingTechnique described herein, and that any or all of these alternateembodiments, as described below, may be used in combination with otheralternate embodiments that are described throughout this document. Notealso that various elements of FIG. 2 were previously introduced in FIG.1, and that any common elements between these two figures share the sameelement numbers.

In general, as illustrated by FIG. 2, the processes enabled by theMotion and Context Sharing Technique begin operation by providing sensorinputs from the sensor pen, touch inputs from the computing device, andcontext information from either or both the sensor pen and computingdevice to the aforementioned gesture activation module. The gestureactivation module then evaluates the available inputs and information totrigger one or more motion gestures, along with any corresponding userinterface (UI).

More specifically, the sensor input module 125 provides sensor inputsreceived from one or more sensors coupled to the sensor pen to thegesture activation module 135. In addition, the touch input module 130provides touch inputs detected by any touch sensitive surface of thecomputing device to the gesture activation module 135. As noted above,virtual “touches” on various projections, surfaces, displays, objects,etc., are also considered by using various well-known techniques totrack user touches on any surface, display or object.

Further, as noted above, in various embodiment, the Motion and ContextSharing Technique also rejects or ignores unwanted or unintendedtouches. An optional palm rejection module 205 is used for this purpose.In particular, the palm rejection module 205 evaluates any touch todetermine whether that touch was intended, and then either accepts thattouch as input for further processing by the touch input module 130, orrejects that touch. In addition, in various embodiments, the palmrejection module 205 disables or ignores (i.e., “rejects”) user toucheson or near particular regions of any touch-sensitive surfaces, dependingupon the context of that touch. Note that “rejected” touches may stillbe handled by the Motion and Context Sharing Technique as an input toknow where the palm is planted, but flagged such that unintentionalbutton presses or gestures will not be triggered in the operating systemor applications by accident.

With respect to context of the computing device and sensor pen, acontext reporting module 210 reports this information to the gestureactivation module 135. Specifically, the context reporting module 210determines the current context of the computing device and/or the sensorpen, and reports that context information to the gesture activationmodule 135 for use in determining motion gestures. Context examplesinclude, but are not limited to, sensor pen and computing deviceindividual or relative motions, whether they are being held, powerstates, application status, etc. Furthermore, the presence or absence(loss of) a pen signal, as well as the signal strength, may be used asan aspect of context as well. For example, in the case of multiple pens,simple triangulation based on signal strengths enables the Motion andContext Sharing Technique to determine approximate relative spatiallocation and proximity of one or more users.

Examples of various motion gestures triggered or activated by thegesture activation module 135 include motion gestures 140 through 165,and motion gestures 215 through 240. Note that the aforementioned userdefined motion gestures 170 are defined via a user interface 245 thatallows the user to define one or more motion gestures using sensor peninputs relative to any combination of touch and context inputs. Each ofthe motion gestures 140 through 165, and motion gestures 215 through 240illustrated in FIG. 2 are described in further detail throughout Section2 of this document, with examples of many of these motion gestures beingillustrated by FIG. 3 through FIG. 10.

2.0 Operational Details of Motion and Context Sharing Technique:

The above-described program modules are employed for implementingvarious embodiments of the Motion and Context Sharing Technique. Assummarized above, the Motion and Context Sharing Technique providesvarious techniques for using a “sensor pen” to enable a variety of inputtechniques based on various combinations of direct-touch inputs andsensor pen inputs that are not restricted to a near-proximity sensingrange of the digitizer of a computing device. The following sectionsprovide a detailed discussion of the operation of various embodiments ofthe Motion and Context Sharing Technique, and of exemplary methods forimplementing the program modules described in Section 1 with respect toFIG. 1 and FIG. 2.

In particular, the following sections provides examples and operationaldetails of various embodiments of the Motion and Context SharingTechnique, including:

-   -   An operational overview of the Motion and Context Sharing        Technique;    -   Exemplary system hardware for implementing the sensor pen;    -   Exemplary interaction techniques using the sensor pen that are        enabled by the Motion and Context Sharing Technique;    -   Exemplary context sensing techniques that consider current        sensor pen context;    -   Correlated finger touch and sensor pen motions for initiating        actions or commands;    -   Active sensor pen motions outside of hover range of the display        for initiating actions or commands;    -   Sensor pen motions combined with direct touch inputs for        initiating actions or commands; and    -   Close range motion gestures of the sensor pen within hover range        or contact with the display.

2.1 Operational Overview:

As noted above, the Motion and Context Sharing Technique-based processesdescribed herein provide various techniques for using a “sensor pen” toenable a variety of input techniques based on various combinations ofdirect-touch inputs and sensor pen inputs that are not restricted to anear-proximity sensing range of the digitizer of a computing device

More specifically, the Motion and Context Sharing Technique considersvarious combinations of sensor pen stroke, pressure, motion, and othersensor pen inputs to enable various hybrid input techniques thatincorporate simultaneous, concurrent, sequential, and/or interleaved,sensor pen inputs and touch inputs (i.e., finger, palm, hand, etc.) ondisplays or other touch sensitive surfaces. This enables a variety ofmotion-gesture inputs relating to the context of how the sensor pen isused or held, even when the pen is not in contact or within sensingrange of the computing device digitizer. In other words, any particulartouch inputs or combinations of touch inputs relative to a computingdevice are correlated with any desired sensor pen inputs, with thosecorrelated inputs then being used to initiate any desired action by thecomputing device.

2.2 Exemplary System Hardware:

In general, the sensor pen includes a variety of sensors, communicationscapabilities, a power supply, and logic circuitry to enable collectionof sensor data and execution of firmware or software instantiated withinmemory accessible to the logic circuitry. For example, in a testedembodiment, the sensor pen was powered by an internal battery, and useda conventional microcontroller device to collect the sensor data and torun firmware. The sensor pen in this tested embodiment included amicro-electro-mechanical systems (MEMS) type three-axis gyroscope, aswell as MEMS type three-axis accelerometer and magnetometer modules.Wireless communications capabilities were provided in the testedembodiment of the sensor pen by an integral 2.4 GHz transceiveroperating at 2 Mbps. Further, in this tested embodiment, the sensor penfirmware sampled the sensors at 200 Hz and wirelessly transmitted sensordata to an associated tablet-based computing device.

Various components of this exemplary sensor pen are discussed herein inthe context of user interaction with either or both the sensor pen and atablet-type computing device (e.g., whether they are being held, moving,power state, application status, etc.) for initiating various motiongestures and other inputs. However, it must be understood that thisexemplary sensor pen implementation is discussed only for purposes ofillustration and explanation, and is not intended to limit the scope orfunctionality of the sensor pen such as types of sensors associated withthe sensor pen, communications capabilities of the sensor pen, etc.Further this exemplary sensor pen implementation is not intended tolimit the scope of functionality of the Motion and Context SharingTechnique or of any motion gestures or other input techniques discussedherein.

2.3 Interaction Techniques:

In general, the Motion and Context Sharing Technique considers a varietyof categories of sensor pen motion and context relative to touch, motionand context of an associated touch sensitive computing device or displaydevice. For purposes of explanation, the following discussion will referto a sketching or drawing type application in the context of atablet-type computing device. However, it should be understood that boththe sensor pen and the Motion and Context Sharing Technique are fullycapable of interaction and interoperation with any desired applicationtype, operating system type, or touch-sensitive computing device.

In the context of the sketching application, a number of semanticallyappropriate mappings for various sensor pen gestures were defined withinthe context of inking and sketching tasks. Further, as noted above, theMotion and Context Sharing Technique also considers the context ofvarious sensor pen gestures relative to the context of the associatedcomputing device, and any contemporaneous user touches of the computingdevice to enable a variety of concurrent pen-and-touch inputs.

Note that in other application contexts, such as, for example, activereading or mathematical sketching, different gestures or mappings can bedefined. In fact, as noted above, any desired user-definable gesturesand concurrent pen-and-touch inputs can be configured for any desiredaction for any desired application, operating system, or computingdevice. The following sections describe various techniques, includingcontext sensing, pen motion gestures away from the display or touchsensitive surface of the computing device, motion gestures combined withtouch input, close-range motion gestures (i.e., within hover range or incontact with the display), and combined sensor pen motions. Further, itshould also be understood that voice or speech inputs can be combinedwith any of the various input techniques discussed herein above toenable a wide range of hybrid input techniques.

2.4 Context Sensing Techniques:

Sensing the motion and resting states of both the stylus and the tabletitself offers a number of opportunities to tailor the user experience tothe context of the user's naturally occurring activity. Several examplesof such techniques are discussed below. Note that the exemplarytechniques discussed in the following paragraphs are provided forpurposes of explanation and are not intended to limit the scope of thesensor pen or the Motion and Context Sharing Technique described herein.

2.4.1 Tool Palette Appears & Disappears with Pen:

On tablet computers, there is often a tension between having toolpalettes and other UI controls on the screen at all times, versusemploying the entire screen for showing the user's content.Advantageously, by considering the context of the user interaction withthe sensor pen and the computing device, the Motion and Context SharingTechnique provides a number of techniques for enhancing user experiencewith respect to such issues.

In the case of a drawing or sketching application, the user interface(UI) tools of interest to the user are often different at differenttimes during any particular session. For example, the UI tools ofinterest while handwriting or sketching may be different than those ofused to browse content or review work-in-progress. Consequently, invarious embodiments, the Motion and Context Sharing Technique considersthe context of the sensor pen to automatically fade in a stylus toolpalette when the user picks up the sensor pen (as determined via one ormore of the aforementioned pen sensors). In the case of a drawingapplication or the like, this tool palette includes tools such as colorchips to change the pen color, as well as controls that change the modeof the stylus (e.g., eraser, highlighter, lasso selection, inking,etc.). Conversely, when the user puts down the pen, or holds the penrelatively still for some period of time, the palette slowly fades outto minimize distraction.

In terms of sensor pen context, picking up or lifting the sensor pen isidentified (via one or more of the sensors) as a transition from a statewhere the pen is not moving (or is relatively still) to a state ofmotion above a fixed threshold. In a tested embodiment of the sensorpen, a multi-axis gyroscope sensor in the pen was used to determinesensor pen motion for this purpose due to the sensitivity of gyroscopesto subtle motions. For example, one technique used by the Motion andContext Sharing Technique identifies pen motion (e.g., picking up orlifting) whenever a three-axis sum-of-squares of gyroscope signalsexceeds some threshold rotational rate (e.g., 36 deg/s). When suchmotion is identified, the Motion and Context Sharing Technique thentriggers the palette to appear or to fade in over the course of someperiod of time (e.g., one second). Conversely, when the pen motion fallsbelow the threshold rotational rate, the palette disappears or fades outover some period of time. In a tested embodiment, palette fade outoccurred over a longer period of time (e.g., five seconds) than palettefade in, and if sensor pen motion resumed before this fade-out finishes,the palette quickly fades back in to full opacity.

For interaction with the palette itself, the Motion and Context SharingTechnique provides various motion gestures relative to the sensor pen toallow the palette UI to respond to either pen taps or finger taps, toallow users to interleave these input modalities to select a current pentool and color. In this context, this enables the Motion and ContextSharing Technique to treat pen and touch inputs interchangeably. Forthis reason, it is also possible to call up the palette using touchalone, by tapping on it with a finger (often useful if the pen iscurrently not in use, for example).

In the context of other application types, picking up or lifting the pencan be used to trigger any desired menu, user interface component, oraction relevant to the particular application being used.

2.4.2 Pen Loss Prevention:

One problem with pens and stylus type devices is that the user can loseor forget the pen. For example, users often leave the stylus on theirdesk, or forget to put it back in the tablet's pen holster after use.Unfortunately, the loss or unavailability of the pen or stylus tends tolimit the use of the user's computing device to a touch-only deviceuntil such time as another pen or stylus can be obtained.

Advantageously, the Motion and Context Sharing Technique addresses thisproblem by sensing the context of the sensor pen relative to thecomputing device, and then automatically reminding or alerting the userwhenever the Motion and Context Sharing Technique observes the tabletmoving away without the sensor pen. Since both the sensor pen and tablet(or other computing device) have motion sensors, and they are incommunication with one another, the Motion and Context Sharing Techniqueevaluates sensor information of the pen and tablet to infer whether ornot they are moving together.

One simple example of this context is that if the tablet starts moving,and continues moving while the pen remains stationary, then a largemessage (e.g., “Forgot the pen?”) appears on the tablet's display,either immediately, or after any desired preset delay. The message maythen optionally then fade away or be dismissed by the user. This servesto remind the user to retrieve the sensor pen. Note that other alerttechniques (e.g., audible, visual, tactile, etc.) on either or both thesensor pen and tablet are used in various embodiments of the Motion andContext Sharing Technique to alert the user to potential sensor penloss. These techniques illustrate how sensing the motion states of boththe tablet and sensor pen can help provide a more complete picture ofthe system's state.

In a related embodiment the Motion and Context Sharing Techniqueconsiders correlated gait patterns, for example, to determine if theuser is walking with both pen and stylus on his person or not. Further,if sensor analysis indicates that the tablet is in a pack, purse, or notvisible to a walking or moving user, an automated alert sent to theuser's mobile phone is initiated in various embodiments to alert theuser to potential sensor pen loss.

2.5 Correlated Finger Touch and Pen Motions:

By evaluating correspondences between touch-screen input (or touchinputs on other surfaces of the computing device) and sensor penmotions, the Motion and Context Sharing Technique infers additionalinformation about how the user is touching the screen or othertouch-sensitive surface. By correlating sensor pen motions with touchinputs, the Motion and Context Sharing Technique enables a variety ofinput scenarios and motion gestures.

For example, if the user touches the screen with the preferred hand(i.e., the same hand that is holding the sensor pen), the Motion andContext Sharing Technique infers via analysis of sensors in either orboth the computing device and the sensor pen that the pen is in motionat the same time that the touch is in motion. FIG. 3 provides a simpleillustration of this scenario. In particular, in this example, theuser's left hand 300 is holding a sensor pen 310, while the index finger305 of that hand is in contact 320 with the surface of the display 330of the computing device 340. Both the index finger 305 and the sensorpen 310 held in the hand are moving in the same direction, asillustrated by the large directional arrows.

In contrast, if the Motion and Context Sharing Technique observes atouch while the sensor pen is relatively stationary, the Motion andContext Sharing Technique infers that the touch was produced by thenon-preferred hand (i.e., touch produced by the hand not holding thepen).

For example, returning to the example of a drawing or sketch typeapplication, in various embodiments, the Motion and Context SharingTechnique provides an input mechanism where a dragging type touch withthe non-preferred hand (i.e., with little or no corresponding penmotion) pans the canvas on the display screen. Conversely, a rubbing ordragging motion on an ink stroke with the preferred hand (e.g., pen istucked between the fingers of that hand, but not in contact with thedisplay or digitizer), digitally smudges the ink. This correlated touchand motion input mimics the action of charcoal artists who repeatedlydraw some strokes, and then tuck the charcoal pencil to blend (smudge)the charcoal with a finger of the same hand.

Clearly, the user may move both hands concurrently. Consequently, evenif the user is holding the pen relatively still, a small amount ofmotion may be sensed by one or more sensors of the sensor pen.Therefore, in various embodiments, the Motion and Context SharingTechnique ignores motions below a dynamically defined threshold byconsidering factors such as relative motion velocity and directionbetween the touch and the sensor pen (i.e., directional correlation). Inaddition, since an initial determination of whether a display pan or inksmudge is intended, in various embodiments the Motion and ContextSharing Technique defers the decision of panning vs. smudging for arelatively short time-window at the onset of a touch gesture while therelative motions are evaluated to make a final determination.

In various embodiments, the dynamically defined motion thresholdincreases when the pen is in rapid motion, and exponentially decaysotherwise. This allows the Motion and Context Sharing Technique tohandle the case where the user has finished smudging, but the pen maynot yet have come to a complete stop, and then switches quickly topanning with the non-preferred hand. The dynamic threshold here helpsthe system to correctly reject the pen motion as a carry-over effectfrom the recently-completed smudging gesture.

In addition, by considering the sensors of both the computing device andsensor pen in a common inertial frame, the Motion and Context SharingTechnique enables various scenarios that consider correspondingdirections of motion (i.e., directional correlation), thereby enablingvery subtle or gentle smudging motions that might otherwise fall below amovement threshold. Advantageously, in the case that the user touchesand starts moving with both hands at approximately the same time, theuse of a common inertial frame enables the Motion and Context SharingTechnique to distinguish which touch point corresponds to the penmotion.

2.6 Active Pen Motion Away from the Display:

In contrast to conventional pen or stylus usage with computing devices,the Motion and Context Sharing Technique allows the sensor pen toprovide input at any distance from the computing device or digitizersurface. Advantageously, the use of sensors coupled to the sensor pen incombination with the communications capabilities of the sensor penenable the Motion and Context Sharing Technique to consider sensor penmotions independent from the computing device. This allows the Motionand Context Sharing Technique to implement various explicit sensorgestures that can be active at all times (or at particular times or inparticular contexts).

One example of an of a motion gesture enabled by considering pen motionsindependently of the computing device “is the aforementioned “roll toundo” gesture, which uses a rolling motion of the pen (i.e., twisting orrotating the pen around the long axis of barrel). Note that this motiongesture is discussed in further detail below. Further, a number of themotion gestures and techniques discussed herein, including variouscontext sensing techniques and various sensor pen motion gesturescombined with direct touch inputs, rely on sensing the motion of the penwhile it is away from the display (i.e., outside of contact and hoverrange of the computing device). Therefore, the ability to sense penactivity at a distance from the display enables many of the sensorpen-motion techniques discussed herein.

2.6.1 Roll to Undo:

In various embodiments, the Motion and Context Sharing Techniqueconsiders sensor pen rolling motions as a distinct gesture for peninput. For example, the aforementioned roll to undo gesture is activatedby user rolling of the sensor around the long axis of the pen, while thesensor pen is beyond hover range of the computing device. In general,rolling of the sensor pen in this manner is detected by sensors such asgyroscopic sensors or accelerometers coupled to the sensor pen. FIG. 4provides a simple illustration of this scenario. In particular, in thisexample, the user's right hand 400 is holding a sensor pen 410, whilethe pen is rotated around the long axis of the sensor pen, asillustrated by the large directional arrow.

In one embodiment, when the Motion and Context Sharing Techniquerecognize this rolling gesture, the Motion and Context Sharing Techniqueautomatically undoes the last action completed by the computing device.Alternately, in a related embodiment, when the Motion and ContextSharing Technique recognize this rolling gesture, a user interface menuappears on the screen that shows the user they have activated the Undocommand. The user can then tap or touch the displayed Undo command oneor more times to undo one or more preceding actions. Advantageously,these embodiments have been observed to speed up user interaction withthe computing device by presenting the Undo command to the user withoutrequiring the user to navigate the application menu structure to locatethe Undo command in a menu and the wasted movement of going to the edgeof the screen to invoke it.

2.6.1 Finger Tap to Redo:

To complement the aforementioned roll to undo motion gesture, the Motionand Context Sharing Technique provides a tap to redo motion gesture thatis activated in response to detection of user finger taps followingactivation of an Undo command. As with the roll to undo gesture, thattap to redo motion gesture can be performed outside the hover range ofthe computing device. Similar to the Undo command, in variousembodiments, a user interface menu appears on the screen that shows theuser they have activated the Redo command. The user can then tap ortouch the displayed Redo command one or more times to redo one or morerecently undone actions.

FIG. 5 provides a simple illustration of this scenario. In particular,in this example, the user's right hand 500 is holding a sensor pen 510,while index finger 520 of that hand taps the barrel of the sensor pen,as illustrated by the large directional arrow. Note that as discussedbelow, a similar input mechanism is used to implement a “barrel tap”input mechanism when the sensor pen is in hover range or contact withthe display of the computing device.

Advantageously, both the roll to undo and finger tap to redo motiongestures interleave stylus motion and touch input in a hybrid inputscenario that takes advantage of the properties of each interactionmodality, while allowing the user to keep the sensor pen close to theuser's working space and, thus, the locus of attention.

2.7 Pen Motion Gestures Combined with Direct Touch:

Combined touch and sensor pen motion gestures provide an additionaltechnique that contrasts with basic sensor pen-motion gestures (e.g.,the roll to undo gesture) by adding a concurrent user touch componentfor activation of various input mechanisms. Advantageously, this allowsthe same sensor pen motion gesture used without touch to be used toinitiate one or more entirely different input mechanisms when that samesensor pen motion is combined with one or more different user touchinputs. However, it should also be understood that there is norequirement for the same sensor pen motion gestures to be used whencombined with various touch inputs.

For example, in addition to the roll to undo technique that interleavespen motion and touch inputs (i.e., pen roll followed by touchingdisplayed undo menu), the Motion and Context Sharing Technique alsoprovides a variety of motion gestures that employ simultaneous,concurrent, sequential touch and pen motion inputs. The combination ofmotion gesture plus touch techniques described below for providingvarious user input mechanisms illustrates how new sensing modalities canbuild on the existing skills and habits of users who may be familiarwith particular applications or type of content.

Examples of these combined motion gestures and touch input include, butare not limited to a “touch and spatter” input mechanism with respect topainting type applications, a “touch and tilt for layers” inputmechanism, and a “touch and roll to rotate” input mechanism. Note that awide range of combined motion gestures and direct touch for initiatingspecific input scenarios or commands is enabled by the Motion andContext Sharing Technique, and that the examples discussed below are notintended to limit the scope of the Motion and Context Sharing Technique.

2.7.1 Touch and Spatter:

Artists working in water media often employ a technique of rapping aloaded brush on the finger to produce spatters of paint on the paper.Such effects can produce natural-looking textures for foliage andlandscapes. In various embodiments, the Motion and Context SharingTechnique provides a corresponding touch and pen-motion gesture thatmimics this physical gesture within the context of a sketching orpainting application.

For example, the touch and spatter input mechanism is initiated when theuser touches the screen with a finger, and then strikes the pen againstthat finger (or other surface) to produce spatters as if the sensor penwere a loaded paintbrush. FIG. 6 provides a simple illustration of thisscenario. In particular, in this example, the index finger 600 of theuser's left hand 610 is touching display 620 of computing device 630. Asensor pen 640 held in the user's right hand 650 is struck against theindex finger 600 to initiate the touch and spatter input mechanism, withthe result being digital paint spatters 660 in a region around the pointwhere the index finger is touching the display 620.

Note that, given the limited hover-sensing range of typical tablets, itis likely that the pen remains out-of-range (e.g., more than ˜1 cm awayfrom the display surface) with respect to hover detection when the userperforms this gesture. Therefore, the tablet does may not know theactual (x, y) location of the pen tip. Consequently, the Motion andContext Sharing Technique produces spatters (in the currently selectedpen color) centered on the finger contact point when sensors in thesensor pen indicate that the user is striking the pen against the fingeror other surface while the user is concurrently touching the display.

More specifically, in a tested embodiment, the Motion and ContextSharing Technique detects an acceleration peak (via accelerometerscoupled to the sensor pen) corresponding to the sensor pen strike. TheMotion and Context Sharing Technique then uses the amplitude of the peakto determine any desired combination of a number and transparency levelof the spatters, how large the individual spatters are, and how far theyscatter from the contact point. Each of these values increase withincreasing acceleration peak amplitudes (corresponding to harder sensorpen strikes). The semi-transparent spatters allow the colors to mix withone another in a natural-looking manner.

Furthermore, to prevent possible unintended spatter activation, invarious embodiments, the Motion and Context Sharing Technique does notrespond to isolated strikes. Instead, in such embodiments, the touch andspatter input mechanism is activated by the user striking the sensor pento finger multiple times to begin the spattering effect. This results ina short delay before the paint spatters begin while ensuring that thespatter effect is actually intended by the user.

2.7.2 Touch and Tilt for Lavers:

A common problem in graphical layout applications (e.g. PowerPoint®,Photoshop®, etc.) is working with multiple layered objects that occludeone another. To address this problem, the Motion and Context SharingTechnique provides a gesture of touching an object and then pitching ortilting the pen backward (or forward) relative to the long axis of thesensor pen to reveal a list of the layered objects in z-order, oralternately, to show or cycle through those layers. The user may thentap on the objects in the list or on any of the displayed layers toselect, reorder, or otherwise interact with the selected layer. Thisinput mechanism is referred to herein as the aforementioned “Touch andTilt for Layers” gesture. In other words, to activate the Touch and Tiltfor Layers gesture, the user touches or holds an object while tiltingthe pen to trigger that input mechanism.

FIG. 7 provides a simple illustration of this scenario. In particular,in this example, the index finger 700 of the user's left hand 710 istouching an area of display 720 of computing device 730 having aplurality of layers 740 or layered objects. A sensor pen 750 held in theuser's right hand 760 is tilted pen backward (or forward) relative tothe long axis of the sensor pen, as illustrated by the large directionalarrow, to reveal and interact with the layers 740 or layered objectscorresponding the point where the index finger 700 is touching thedisplay 720.

Note that while the pen pitching or tilting motion can be used toactivate the touch and tilt for layers gesture without a concurrenttouch, it was observed that arbitrary user tilting motions of the sensorpen sometimes inadvertently activated the touch and tilt for layersgesture. Consequently, in various embodiments, the Motion and ContextSharing Technique limits activation of the touch and tilt for layersgesture to contexts where the holds a finger on a stack of objects orother layers to avoids inadvertent activation of layer cycling orinteractions. Therefore, in this example, the touch component of thegesture serves a double purpose. First, touching the screen activatesthe pen pitching motion for recognition. Second, touching the screenalso identifies which objects or layer stacks the touch and tilt forlayers gesture applies to.

To further limit potential inadvertent activations, in closely relatedembodiments, the touch and tilt for layers gesture was activated by atouch concurrent with tilting the pen away from the screen and then backtowards the screen (or the opposite motions, i.e., towards and thenaway), within a limited time-window.

2.7.3 Touch and Roll to Rotate:

A related input mechanism, referred to herein as the aforementionedtouch and roll to rotate gesture is activated by holding or touching anobject on the display surface while concurrently rolling or twisting thesensor pen around the long axis of the sensor pen. Note that thisgesture contrasts with the “roll to undo” gesture discussed above inthat when the Motion and Context Sharing Technique infers a concurrenttouch in combination with the rolling motion, the touch and roll torotate gesture is activated instead of the roll to undo gesture (whichoccurs when there is no concurrent touch).

In a tested embodiment of the Motion and Context Sharing Technique, whenthe user touches an object and rolls the pen, this enables a rotationmode. In one embodiment, this rotation mode is implemented by allowingthe user to dial his finger (i.e., move the finger in a curving motionon the display surface) to precisely rotate the object. In a relatedembodiment, rotation is controlled by either continuing to rotate thesensor pen, or by contacting the sensor pen on the display surface, andusing the sensor pen in a manner similar to the finger dialing motionnoted above.

FIG. 8 provides a simple illustration of the touch and roll to rotateinput mechanism. In particular, in this example, the index finger 800 ofthe user's left hand 810 is touching an area of display 820 of computingdevice 830 having an object 840. A sensor pen 850 held in the user'sright hand 860 is rotated around the long axis of the sensor pen, asillustrated by the large directional arrow to initiate the touch androll to rotate input mechanism. The selected object 840 is then rotatedaround its axis of rotation (illustrated by the large directional arrowaround object 840) using either the user's finger, or additionalrotation motions of the sensor pen 850.

2.8 Close-Range Motion Gestures Combined with Hover or Contact:

Similar to touch combined with sensor pen motions, the Motion andContext Sharing Technique provides additional input mechanisms thatcombine direct sensor pen contact or hover (i.e., sensor pen in hoverrange of the digitizer of the computing device). For example, inputmechanisms implemented by combining sensor pen hover with sensor penmotion gestures (determined via one or more of the sensors coupled tothe sensor pen) include, but are not limited to a “vertical menu” inputmechanism and a “barrel tap” input mechanism, both of which use thesensed (x, y) location of the pen tip (via hover evaluation of the penby the computing device) to determine where on the display to bring up amenu. Another input mechanism, referred to herein as a “hard stroke”input mechanism, combines direct contact of the sensor pen with thedisplay in combination with sensor pen motions. Each of these exemplaryinput mechanisms are discussed in further detail below.

2.8.1 Vertical Menu:

In various embodiments, the vertical menu input mechanism is initiatedby using various sensors of the pen to detect a user initiated motion ofthe sensor pen that occurs with the pen coming towards or into proximityrange (or in contact with the screen) in an orientation approximatelyperpendicular relative to the display screen of the computing device. Inother words, holding the sensor pen approximately perpendicular relativeto the display (e.g., approximately vertical pen pose when the computingdevice is lying flat), and either approaching or within hover rangerelative to the display screen of the computing device, activates a UIwindow at or near the sensed (x, y) location of the pen tip, therebyenabling efficient interleaving of stroke input with menu invocation.

Note that in various embodiments, a timing factor is also considered forinitiating the vertical menu. For example, in such embodiments, thevertical menu is initiated when the sensor pen is held approximatelyperpendicular relative to the display and approximately stationary for ashort time (e.g., a fixed or adjustable time threshold) within the hoverrange of the display. Note that in various embodiments, the UI window ofthe vertical menu is initiated when the pen is an approximatelyperpendicular pose, for a certain amount of time, as it approaches thedisplay, regardless of whether the pen is in a proximity or hover rangeof the display. Consequently, it should be clear that in suchembodiments, the Motion and Context Sharing Technique indirectly usesthe pen's ability to know its orientation even when it is beyond thesensing range of the computing device to successfully trigger thevertical menu.

When initiated or activated, the vertical menu input mechanism triggersa UI mode that brings up a localized (relative to the pen tip) UI menu.Consequently, bringing the pen close to the screen with the pen in thisrelative perpendicular pose will initiate opening or expansion of avertical software menu or the like. Conversely, motions extending awayfrom the computing device or display (i.e., motions moving away from thescreen or beyond the proximity sensing range) may initiate closing orcontraction of the vertical software menu or the like. Other mechanismsfor closing the UI menu include, but are not limited to, automaticallyclosing the menu after the user picks a command from the UI menu, inresponse to taps (pen or finger) somewhere else on the screen, detectionof some other motion gesture, etc.

Advantageously, menus that appear at or near the locus of interaction(i.e., near the pen tip) can save the user from round-trips with thesensor pen (or other pointing device) to tool palettes or other menus atthe edge of the screen. Such localized menus are particularly useful forfrequent commands, as well as contextual commands such as Copy and Pastethat integrate object selection and direct manipulation with commands.Similarly, in various embodiments, the Motion and Context SharingTechnique initiates context sensitive menus, application popups, etc.,when activated by the vertical menu input mechanism.

In various embodiments, the vertical menu input mechanism initiates amarking menu when the pen is held approximately perpendicular relativeto the display and approximately stationary for a short time within thehover range of the display. Note that marking menus are also sometimesreferred to as “pie menus” or “radial menus.” In general, the markingmenu provides a generally circular context menu where selection of amenu item depends on direction. Marking menus may be made of several“pie slices” or radially arranged menu options or commands around acenter that may be active or inactive. Each such slice may include anynumber of menu items (similar to a list or set of menu items in aconventional drop down menu). Further, in various embodiments, commandsor menu items in the marking menu are user-definable, and of course,vertical menus could be employed with other well-known types of commandselection techniques, such as drawing gestures or picking fromtraditional pull-down menus as well. In this sense, the “vertical menu”can be conceived as a general purpose mode-switching technique, whichenables the pen to input commands, gestures, or perform other taskssecondary to a primary “inking” state.

In one embodiment, the marking menu is initiated as simply a display ofdirections (e.g., arrows, lines, icons, etc.), radiating out from acenter of the marking menu, in which the sensor pen can be stroked toinitiate commands without displaying a visible menu of commands. Oncethis marking menu is displayed, the user may then bring the stylus orsensor pen into contact with the display and stroke in any of thedisplayed directions to invoke a command. In various embodiments, if theuser continues to hold the pen relatively stationary for a short periodof time, the Motion and Context Sharing Technique initiates marking menupops up that reveal a mapping of stroke direction to command. Thevertical menu input mechanism thus combines stylus motion and pen-tipstroke input in the same technique.

In various embodiments, the vertical menu input mechanism is activatedrelative to a particular object when the user places the sensor pen overthe object in a pose that is approximately perpendicular relative to thedisplay. In this case, the marking menu includes object-specificcommands (e.g. copy, paste, etc.) that take the current object andscreen location as operands. Advantageously, this integrates objectselection and command invocation into a single fluid pen gesture ratherthan requiring the user to perform a sequence of actions to achieve thesame result.

In operation, the Motion and Context Sharing Technique uses anapproximately perpendicular posture of the sensor pen relative to thedisplay to trigger the vertical menu input mechanism from thehover-state. The approximately perpendicular posture of the sensor penrelative to the display is detected using a combination of theaccelerometer and gyroscope sensors coupled to the sensor pen relativeto a sensed orientation of the display. The accelerometer is used toestimate the orientation of the pen, but even if the accelerometerindicates that the pen is relatively near-perpendicular, this sensorreading will not trigger the vertical menu input mechanism when thegyroscope indicates that the pen is still moving beyond some thresholdamount. In this way, the Motion and Context Sharing Technique avoidsfalse positive activation of the marking menu if the user briefly passesthrough an approximately perpendicular relative pose while handling thesensor pen.

FIG. 9 provides a simple illustration of a vertical menu input scenario.In particular, in this example, a sensor pen 900 held in the user'sright hand 910 is in hover range above the display 920 of computingdevice 930. Note that hover is indicated in this figure by the cone 940shown by broken lines extending from the tip of the sensor pen 900. Byholding the sensor pen 900 approximately vertical within hover range ofthe display 920, the Motion and Context Sharing Technique initiates amarking menu 950, which in this example includes menu options or commandchoices A through F. The user selects or initiates any of these commandsby either touching one of the displayed commands, sweeping or strokingthe sensor pen 900 in the direction of one of the displayed commands, orcontacting one of the displayed commands with the sensor pen.

2.8.2 Barrel Tap:

One mechanism by which the sensor pen can implement button presses is tosimply include one or more buttons or the like on the barrel of thesensor pen, with button presses then being communicated by the pen tothe computing device. However, since the sensor pen includes a varietyof sensors, the Motion and Context Sharing Technique instead uses one ormore of the existing sensors to identify user finger taps on the barrelof the sensor pen to initiate a barrel tap input mechanism. This barreltap input mechanism can be used for any of a variety of purposes,including, but not limited to, button press emulations.

In general, the barrel tap input mechanism is activated by sensingrelatively hard-contact finger taps on the barrel of the pen as a way to“replace” mechanical button input. In a tested embodiment, the Motionand Context Sharing Technique evaluates accelerometer data to identifyan acceleration spike approximately perpendicular to the long axis ofthe sensor pen while the pen is approximately stationary in the hoverstate or in direct contact with the display. Activation of the barreltap input mechanism brings up a menu with commands specific to theobject that the user hovers over or contacts with the sensor pen, withthat menu being approximately centered under the pen tip.

Note that the barrel tap input mechanism differs from the aforementioned“finger tap” input mechanism in that the finger tap mechanism isperformed without a concurrent touch or hover relative to the computingdevice, while the barrel tap input mechanism is performed while the penis in hover range or contact within a displayed object.

2.8.3 Hard Tap and Hard Draq Input Mechanisms:

In various embodiments, the Motion and Context Sharing Techniqueprovides additional input mechanisms initiated by the sensor pen thatare roughly analogous to “hard tap” and “hard drag” techniques that beenimplemented using existing finger touch inputs.

However, rather than using the finger to provide touch inputs, theMotion and Context Sharing Technique instead evaluates accelerometerthresholds to determine whether the user intends a hard tap or a harddrag input using the sensor pen. In a tested embodiment in a drawingtype application, bringing the stylus or sensor pen down relatively hardon the display surface results in a hard tap input mechanism thattriggers a lasso mode, whereas relatively softer sensor pen strokes onthe display surface result in a hard drag input mechanism that producesdigital ink. Adjustable or customizable accelerometer thresholds fordistinguishing the “hard” contact from softer strokes are used by theMotion and Context Sharing Technique. Further, in various embodiments,the angle at which a user is holding the pen when it comes into contactwith the display is considered by the Motion and Context SharingTechnique for differentiating between the two input mechanisms.

FIG. 10 provides a simple illustration of the hard tap input mechanism.In particular, in this example, a sensor pen 1000 held in the user'sright hand 1010 is brought down relatively hard onto a point 1020 of thedisplay 1030 of computing device 1040. The resulting contact of thesensor pen 1000 onto the surface of display 1030 (identified usingaccelerometers of the sensor pen) initiates the hard tap input mechanismthat triggers the aforementioned lasso mode. The user then uses either afinger touch or drags the sensor pen 1000 across the surface of thedisplay 1030 to draw lasso outlines for use in selecting objects,regions, etc., in the drawing type application.

3.0 Operational Summary:

The processes described above with respect to FIG. 1 through FIG. 10,and in further view of the detailed description provided above inSections 1 and 2, are illustrated by the general operational flowdiagram of FIG. 11. In particular, FIG. 11 provides an exemplaryoperational flow diagram that summarizes the operation of some of thevarious embodiments of the Motion and Context Sharing Technique. Notethat FIG. 11 is not intended to be an exhaustive representation of allof the various embodiments of the Motion and Context Sharing Techniquedescribed herein, and that the embodiments represented in FIG. 11 areprovided only for purposes of explanation.

Further, it should be noted that any boxes and interconnections betweenboxes that are represented by broken or dashed lines in FIG. 11represent optional or alternate embodiments of the Motion and ContextSharing Technique described herein, and that any or all of theseoptional or alternate embodiments, as described below, may be used incombination with other alternate embodiments that are describedthroughout this document.

In general, as illustrated by FIG. 11, the Motion and Context SharingTechnique begins operation by receiving 1100 sensor input from one ormore sensors (1105 through 1150) of the sensor pen. These inputs arethen transmitted 1155 from the sensor pen to the touch-sensitivecomputing device. The Motion and Context Sharing Technique also receives1160 one or more touch inputs from the touch-sensitive computing device.Further, the Motion and Context Sharing Technique also optionallyreceives 1165 context information from either or both the sensor pen andthe context sensitive computing device.

The Motion and Context Sharing Technique then evaluates 1170simultaneous, concurrent, sequential, and/or interleaved sensor peninputs and touch inputs relative to contexts of sensor pen and computingdevice. This evaluation serves to identify one or more motion gestures1180 corresponding to the various sensor, touch and context inputs. TheMotion and Context Sharing Technique then automatically initiates 1175one or more motion gestures 1180 based on the evaluation. Finally, asnoted above, in various embodiments, a UI or the like is provided 1185for use in defining and/or customizing one or more of the motiongestures 1180.

4.0 Exemplary Operating Environments:

The Motion and Context Sharing Technique described herein is operationalwithin numerous types of general purpose or special purpose computingsystem environments or configurations. FIG. 12 illustrates a simplifiedexample of a general-purpose computer system in combination with astylus or pen enhanced with various sensors with which variousembodiments and elements of the Motion and Context Sharing Technique, asdescribed herein, may be implemented. It should be noted that any boxesthat are represented by broken or dashed lines in FIG. 12 representalternate embodiments of the simplified computing device and sensor pen,and that any or all of these alternate embodiments, as described below,may be used in combination with other alternate embodiments that aredescribed throughout this document.

For example, FIG. 12 shows a general system diagram showing a simplifiedtouch-sensitive computing device 1200. In general, such touch-sensitivecomputing devices 1200 have one or more touch-sensitive surfaces 1205 orregions (e.g., touch screen, touch sensitive bezel or case, sensors fordetection of hover-type inputs, optical touch sensors, etc.). Examplesof touch-sensitive computing devices 1200 include, but are not limitedto, touch-sensitive display devices connected to a computing device,touch-sensitive phone devices, touch-sensitive media players,touch-sensitive e-readers, notebooks, netbooks, booklets (dual-screen),tablet type computers, or any other device having one or moretouch-sensitive surfaces or input modalities.

To allow a device to implement the Motion and Context Sharing Technique,the computing device 1200 should have a sufficient computationalcapability and system memory to enable basic computational operations.In addition, the computing device 1200 may include one or more sensors1210, including, but not limited to, accelerometers, cameras, capacitivesensors, proximity sensors, microphones, multi-spectral sensors, pen orstylus digitizer, etc. As illustrated by FIG. 12, the computationalcapability is generally illustrated by one or more processing unit(s)1225, and may also include one or more GPUs 1215, either or both incommunication with system memory 1220. Note that the processing unit(s)1225 of the computing device 1200 of may be specialized microprocessors,such as a DSP, a VLIW, or other micro-controller, or can be conventionalCPUs having one or more processing cores, including specializedGPU-based cores in a multi-core CPU.

In addition, the computing device 1200 may also include othercomponents, such as, for example, a communications interface 1230 forreceiving communications from sensor pen device 1235. The computingdevice 1200 may also include one or more conventional computer inputdevices 1240 or combinations of such devices (e.g., pointing devices,keyboards, audio input devices, voice or speech-based input and controldevices, video input devices, haptic input devices, touch input devices,devices for receiving wired or wireless data transmissions, etc.). Thecomputing device 1200 may also include other optional components, suchas, for example, one or more conventional computer output devices 1250(e.g., display device(s) 1255, audio output devices, video outputdevices, devices for transmitting wired or wireless data transmissions,etc.). Note that typical communications interfaces 1230, input devices1240, output devices 1250, and storage devices 1260 for general-purposecomputers are well known to those skilled in the art, and will not bedescribed in detail herein.

The computing device 1200 may also include a variety of computerreadable media. Computer readable media can be any available media thatcan be accessed by computer device 1200 via storage devices 1260 andincludes both volatile and nonvolatile media that is either removable1270 and/or non-removable 1280, for storage of information such ascomputer-readable or computer-executable instructions, data structures,program modules, or other data. By way of example, and not limitation,computer readable media may comprise computer storage media andcommunication media. Computer storage media refers to tangible computeror machine readable media or storage devices such as DVD's, CD's, floppydisks, tape drives, hard drives, optical drives, solid state memorydevices, RAM, ROM, EEPROM, flash memory or other memory technology,magnetic cassettes, magnetic tapes, magnetic disk storage, or othermagnetic storage devices, or any other device which can be used to storethe desired information and which can be accessed by one or morecomputing devices.

Storage of information such as computer-readable or computer-executableinstructions, data structures, program modules, etc., can also beaccomplished by using any of a variety of the aforementionedcommunication media to encode one or more modulated data signals orcarrier waves, or other transport mechanisms or communicationsprotocols, and includes any wired or wireless information deliverymechanism. Note that the terms “modulated data signal” or “carrier wave”generally refer a signal that has one or more of its characteristics setor changed in such a manner as to encode information in the signal. Forexample, communication media includes wired media such as a wirednetwork or direct-wired connection carrying one or more modulated datasignals, and wireless media such as acoustic, RF, infrared, laser, andother wireless media for transmitting and/or receiving one or moremodulated data signals or carrier waves. Combinations of the any of theabove should also be included within the scope of communication media.

Retention of information such as computer-readable orcomputer-executable instructions, data structures, program modules,etc., can also be accomplished by using any of a variety of theaforementioned communication media to encode one or more modulated datasignals or carrier waves, or other transport mechanisms orcommunications protocols, and includes any wired or wireless informationdelivery mechanism. Note that the terms “modulated data signal” or“carrier wave” generally refer to a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. For example, communication media includes wired mediasuch as a wired network or direct-wired connection carrying one or moremodulated data signals, and wireless media such as acoustic, RF,infrared, laser, and other wireless media for transmitting and/orreceiving one or more modulated data signals or carrier waves.Combinations of the any of the above should also be included within thescope of communication media.

Further, software, programs, and/or computer program products embodyingthe some or all of the various embodiments of the Motion and ContextSharing Technique described herein, or portions thereof, may be stored,received, transmitted, or read from any desired combination of computeror machine readable media or storage devices and communication media inthe form of computer executable instructions or other data structures.

Finally, the Motion and Context Sharing Technique described herein maybe further described in the general context of computer-executableinstructions, such as program modules, being executed by a computingdevice. Generally, program modules include routines, programs, objects,components, data structures, etc., that perform particular tasks orimplement particular abstract data types. The embodiments describedherein may also be practiced in distributed computing environments wheretasks are performed by one or more remote processing devices, or withina cloud of one or more devices, that are linked through one or morecommunications networks. In a distributed computing environment, programmodules may be located in both local and remote computer storage mediaincluding media storage devices. Still further, the aforementionedinstructions may be implemented, in part or in whole, as hardware logiccircuits, which may or may not include a processor.

The sensor pen device 1235 illustrated by FIG. 12 shows a simplifiedversion of a pen or stylus augmented with pen sensors 1245, logic 1265,a power source 1275, and basic I/O capabilities 1285. As discussedabove, examples of pen sensors 1245 for use with the sensor pen device1235 include, but are not limited to, inertial sensors, accelerometers,pressure sensors, grip sensors, near-field communication sensors, RFIDtags and/or sensors, temperature sensors, microphones, magnetometers,capacitive sensors, gyroscopes, etc.

In general, the logic 1265 of the sensor pen device 1235 is similar tothe computational capabilities of computing device 1200, but isgenerally less powerful in terms of computational speed, memory, etc.However, the sensor pen device 1235 can be constructed with sufficientlogic 1265 such that it can be considered a standalone capablecomputational device.

The power source 1275 of the sensor pen device 1235 is implemented invarious form factors, including, but not limited to, replaceablebatteries, rechargeable batteries, capacitive energy storage devices,fuel cells, etc. Finally, the I/O 1285 of the sensor pen device 1235provides conventional wired or wireless communications capabilities thatallow the sensor pen device to communicate sensor data and otherinformation to the computing device 1200.

The foregoing description of the Motion and Context Sharing Techniquehas been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the claimed subject matterto the precise form disclosed. Many modifications and variations arepossible in light of the above teaching. Further, it should be notedthat any or all of the aforementioned alternate embodiments may be usedin any combination desired to form additional hybrid embodiments of theMotion and Context Sharing Technique. It is intended that the scope ofthe invention be limited not by this detailed description, but rather bythe claims appended hereto. Although the subject matter has beendescribed in language specific to structural features and/ormethodological acts, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as example forms of implementingthe claims.

What is claimed is:
 1. A computer-implemented process, comprising usinga computer to perform process actions for: receiving sensor inputs froma plurality of sensors coupled to a stylus device; detecting zero ormore finger touches on any touch sensitive surface of a touch-sensitivecomputing device; evaluating the sensor inputs in combination with anyfinger touches to identify an intended motion gesture from a pluralityof available motion gestures; and initiating a user interface actioncorresponding to the intended motion gesture in an application executingon the touch-sensitive computing device.
 2. The computer-implementedprocess of claim 1 wherein one or more of the sensors are used todetermine motion of the stylus device relative to motion of thetouch-sensitive computing device, and wherein those motions arecorrelated for use in identifying the intended motion gesture.
 3. Thecomputer-implemented process of claim 1 further comprising initiating analert on the touch-sensitive computing device when one or more sensorscoupled to the touch-sensitive computing device indicate that thetouch-sensitive computing device is moving away from the stylus device.4. The computer-implemented process of claim 1 wherein one or more ofthe sensors are used to determine orientation of the stylus devicerelative to the touch-sensitive computing device, and wherein thatrelative orientation is used to identify the intended motion gesture. 5.The computer-implemented process of claim 1 wherein the stylus device isoutside of a hover range of the touch-sensitive computing device whilereceiving the sensor inputs from the plurality of sensors coupled to thestylus device.
 6. The computer-implemented process of claim 1 furthercomprising a user interface for specifying sensor inputs and touchescorresponding to one or more of the plurality of available motiongestures.
 7. The computer-implemented process of claim 1 wherein one ormore of the sensors indicate a rotation of the stylus device around along axis of the stylus device concurrently with a touch of an objectdisplayed on a touch screen of the touch-sensitive computing device, andwherein the intended motion gesture corresponds to a user interfaceaction for rotating the touched object.
 8. The computer-implementedprocess of claim 1 wherein one or more of the sensors indicate that thestylus device is within hover range and in an approximatelyperpendicular orientation relative to a touch screen of thetouch-sensitive computing, and wherein the intended motion gesturecorresponds to a user interface action for activating a marking menu onthe display device.
 9. The computer-implemented process of claim 1wherein one or more of the sensors indicate a rotation of the stylusdevice around a long axis of the stylus device when the stylus device isbeyond hover range of a touch screen of the touch-sensitive computingdevice, and wherein the intended motion gesture corresponds to a userinterface action for undoing a prior action of the application executingon the touch-sensitive computing device.
 10. The computer-implementedprocess of claim 1 wherein one or more of the sensors are used todetermine when the stylus device is picked up from a surface, andwherein the intended motion gesture corresponds to a user interfaceaction for presenting a context sensitive menu relating to a currentlyactive object in the application executing on the touch-sensitivecomputing device.
 11. The computer-implemented process of claim 1wherein one or more of the sensors are used to determine motion of thestylus device and motion of a touch on the touch-sensitive computingdevice, and when the those motions are in generally the same direction,using the directional correlation of those motions to identify theintended motion gesture.
 12. A method for initiating actions in anapplication executing on a touch-sensitive computing device, comprising:receiving sensor inputs from a plurality of sensors coupled to a stylusdevice relative to a touch-sensitive computing device; detecting zero ormore finger touches on any touch sensitive surface of thetouch-sensitive computing device; evaluating the sensor inputs incombination with any finger touches to identify an intended motiongesture from a plurality of available motion gestures; and initiating auser interface action corresponding to the intended motion gesture in anapplication executing on the touch-sensitive computing device.
 13. Themethod of claim 12 wherein one or more of the sensors indicate tappingon a barrel of the stylus device when the stylus device is within hoverrange of the touch-sensitive computing device, and wherein the intendedmotion gesture corresponds to a user interface action for emulating abutton press of an input device.
 14. The method of claim 12 wherein oneor more of the sensors indicate tapping on a barrel of the stylus devicewhen the stylus device is beyond hover range of the touch-sensitivecomputing device, and wherein the intended motion gesture corresponds toa user interface action for redoing a prior action of the applicationexecuting on the touch-sensitive computing device.
 15. The method ofclaim 12 wherein one or more of the sensors indicate a tilting of thestylus device relative to a display of the touch-sensitive computingdevice, and wherein a finger touch on the touch-sensitive computingdevice corresponds to two or more layers objects, and wherein theintended motion gesture corresponds to a user interface action forcycling between the layers.
 16. The method of claim 12 wherein theapplication executing on the touch-sensitive computing device is apainting type application and wherein one or more of the sensorsindicate rapping of the stylus device against a surface while there is atouch on a display of the touch-sensitive computing device, and whereinthe intended motion gesture corresponds to a user interface action fordigitally spattering a plurality of digital paint drops around the touchon the display.
 17. A computer storage media having computer executableinstructions stored therein, said instructions causing a computingdevice to execute a method comprising: receiving sensor inputs from aplurality of sensors coupled to a stylus device relative to atouch-sensitive computing device; detecting zero or more finger toucheson any touch sensitive surface of the touch-sensitive computing device;evaluating the sensor inputs in combination with any finger touches toidentify an intended motion gesture from a plurality of available motiongestures; and initiating a user interface action corresponding to theintended motion gesture in an application executing on thetouch-sensitive computing device.
 18. The computer storage media ofclaim 17 wherein one or more of the sensors are used to determine motionof the stylus device relative to motion of the touch-sensitive computingdevice, and wherein those motions are correlated for use in identifyingthe intended motion gesture.
 19. The computer storage media of claim 17further comprising initiating an alert on the touch-sensitive computingdevice when one or more sensors coupled to the touch-sensitive computingdevice indicate that the touch-sensitive computing device is moving awayfrom the stylus device.
 20. The computer storage media of claim 17wherein one or more of the sensors are used to determine orientation ofthe stylus device relative to the touch-sensitive computing device, andwherein that relative orientation is used to identify the intendedmotion gesture.